Generating of 3d-printed custom wearables

ABSTRACT

Disclosed is a platform for generating and delivering 3-D printed wearables. The platform includes scanning, image processing, machine learning, computer vision, and user input to generate a printed wearable. Scanning occurs in a number of ways across a number of devices. The variability of scanning generates a number of scanning output types. Outputs from the scanning process are normalized into a single type during image processing. The computer vision and machine learning portions of the platform use the normalized body scan to develop models that may be used by a 3D printer to generate a wearable customized to the user. The platform further provides opportunities for the user to check the work of the scanning, image processing, computer vision, and machine learning. The user input enables the platform to improve and inform the machine learning aspects.

CLAIM FOR PRIORITY

This application is a continuation of U.S. patent application Ser. No.15/390,406, entitled, “Generation of 3D-Printed Custom Wearables,” filedon Dec. 23, 2016, which claims benefit to U.S. Provisional PatentApplication No. 62/275,242, entitled “Generation of 3D-Printed CustomWearables,” filed on Jan. 6, 2016. The contents of the above-identifiedapplications are incorporated by reference in their entirety.

TECHNICAL FIELD

This disclosure relates to 3-D digital modeling and subsequent 3-Dprinting. This disclosure more particularly relates to input and outputhandling of image data to generate 3-D models for printing.

BACKGROUND

3-D printing can be used to create customized items. However, thepresent cost of ownership of a 3-D printer and the requisite skillrequired for proficient use is prohibitive for most people. Generationof 3-D models for printing that fit a particular wearer, as opposed tostandardized sizing schemes, adds a further complexity. There are anumber of ways to capture body data; however, going from a number ofdisparate data types to a usable 3-D model for 3-D printing in anaccessible fashion is also a difficult problem.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram illustrating a system for the generation ofcustomized 3D printed wearables;

FIG. 2 is a flowchart illustrating a process for generating custom 3Dprinted wearables;

FIG. 3 is a flowchart illustrating a process for acquiring images of theuser and other data to be used in creating a customized 3D printedwearable;

FIG. 4 is a flowchart illustrating a process by which the mobile deviceinteracts with the user to acquire images of the user;

FIG. 5 is a flowchart illustrating a process for performing computervision on collected images of a user;

FIG. 6 is an illustration of a coordinate graph including a collectionof X,Y locations along a body curve;

FIG. 7 is a flowchart illustrating a process for customizingtessellation models;

FIG. 8 is an illustration of a negative normal technique to createcomplex tessellation files;

FIG. 9 is a block diagram illustrating a distributed system for thegeneration of customized 3D printed wearables;

FIG. 10 is a flowchart illustrating API access at a number of steps inwearable generation;

FIG. 11 is a flowchart illustrating an embodiment for handling a numberof input body data types;

FIG. 12 is a flowchart illustrating wearable generation includingsimultaneous computer vision and machine learning processes;

FIG. 13 is a flowchart illustrating distance measurement in images takenfrom single lens 2-D cameras;

FIG. 14 is a graphic illustration of a customized computer mousegenerated through body scanning;

FIG. 15 is a graphic illustration of a customized ear headphonegenerated through body scanning;

FIG. 16 is a graphic illustration of an assortment of wearables that aregenerated via body imaging followed by 3-D printing;

FIG. 17 is a graphic illustration of customized eye glasses generatedthrough body scanning; and

FIG. 18 is a graphic illustration of a customized brace generatedthrough body scanning.

DETAILED DESCRIPTION

To generate 3D printed wearable objects (“wearables”) with simpleinstructions and minimal processing, the technique introduced hereenables commonly available mobile devices to be used to image theprospective wearer and enables a processing server or distributed set ofservers to use the resulting images to generate of a tessellation model.These tessellation model can then be used to generate a 3D printedwearable object (hereinafter simply “wearable”). The term “wearable”refers to articles, adornments or items designed to be worn by a user,incorporated into another item worn by a user, act as an orthosis forthe user, or interfacing with the contours of a user's body. An exampleof a wearable used throughout this disclosure to facilitate descriptionis a shoe insole. A shoe insole is illustrative in that the shape andstyle one particular person would want over the shape and style anotherperson would want tend to vary greatly across people, and customizationis an important detail. Nonetheless, the teachings in this disclosureapply similarly to other types of wearables, such as bracelets, rings,bras, helmets, earphones, goggles, support braces (e.g. knee, wrist),gauge earrings, and body-contoured peripherals.

FIG. 1 is a block diagram illustrating a system for the generation ofcustomized 3D printed wearables 20. Included in the system is thecapability for providing body part input data. Provided as a firstexample of such a capability in FIG. 1 is a mobile processing devicethat includes a digital camera and is equipped to communicate overwireless network, such as a smartphone, tablet computer, a networkeddigital camera or other suitable known mobile devices in the art(hereafter, “mobile device”) 22, a processing server 24, and a 3Dprinter 26. The system further can include a manual inspection computer28.

The mobile device 22 is a device that is capable of capturing andtransmitting images over a wireless network, such as the Internet 30. Inpractice, a number of mobile devices 22 can be used. In someembodiments, mobile device 22 is a handheld device. Examples of mobiledevice 22 include a smart phone (e.g., Apple iPhone, Samsung Galaxy), aconfocal microscopy body scanner, an infrared camera, a ultrasoundcamera, a digital camera, and tablet computer (e.g., Apple iPad or DellVenture 8 7000). The mobile device 22 is a processor enabled deviceincluding a camera 34, a network transceiver 36A, a user interface 38A,and digital storage and memory 40A containing client applicationsoftware 42.

The camera 34 on the mobile device may be a simple digital camera or amore complex 3D camera, scanning device, or video capture device.Examples of 3D cameras include Intel RealSense cameras or Lytro lightfield cameras. Further examples of complex cameras may include scannersdeveloped by TOM-CAT Solutions, LLC (the TOM-CAT, or iTOM-CAT), adaptedversions of infrared cameras, ultrasound cameras, or adapted versions ofintra-oral scanners by 3 Shape.

Simple digital cameras (including no sensors beyond 2-D optical) usereference objects of known size to calculate distances within images.Use of a 3D camera may reduce or eliminate the need for a referenceobject because 3D cameras are capable of calculating distances within agiven image without any predetermined sizes/distances in the images.

The mobile device also provides a user interface 38A that is used inconnection with the client application software 42. The clientapplication software 42 provides the user with the ability to selectvarious 3D printed wearable products. The selection of productscorresponds with camera instructions for images that the user is tocapture. Captured images are delivered over the Internet 30 to theprocessing server 24.

Processor 32B operates processing server 24. The processing server 24receives image data from the mobile device 22. Using the image data,server application software 44 performs image processing, machinelearning and computer vision operations that populate characteristics ofthe user. The server application software 44 includes computer visiontools 46 to aid in the performance of computer vision operations.Examples of computer vision tools 46 include OpenCV or SimpleCV, thoughother suitable examples are known in the art and may be programmed toidentify pixel variations in digital images. Pixel variation data isimplemented as taught herein to produce desired results.

In some embodiments, a user or administrative user may perform manualchecks and/or edits to the results of the computer vision operations.The manual checks are performed on the manual inspection computer 28 orat a terminal that accesses processing server 24's resources. Theprocessing server 24 includes a number of premade tessellation modelkits 48 corresponding to products that the user selects from the clientapplication software 42. Edits may affect both functional and cosmeticdetails of the wearable —edits include looseness/tightness, and highrise/low rise fit. Edits are further stored by the processing server 24as observations to improve machine learning algorithms.

In some embodiments, the tessellation model kits 48 are used as astarting point from which the processing server 24 appliescustomizations. Tessellation model kits 48 are a collection of datafiles that can be used to digitally render an object for 3D printing andto print the object using the 3D printer 26. Common file types oftessellation model kits 48 include 0.3dm, 0.3ds, .blend, .bvh, .c4d,.dae, .dds, .dxf, .fbx, .lwo, .lws, .max, .mtl, .obj, .skp, .stl, .tga,or other suitable file types known in the art. The customizationsgenerate a file for use with a 3D printer. The processing server 24 isin communication with the 3D printer 26 in order to print out the user'sdesired 3D wearable. In some embodiments, tessellation files 48 aregenerated on the fly from the input provided to the system. Thetessellation file 48 is instead generated without premade input throughan image processing, computer vision, and machine learning process.

Numerous models of 3D printer 26 may be used by the invented system. 3Dprinters 26 vary in size of printed article. Based on the type of 3Dwearable users are selecting, varying sizes of 3D printer 26 areappropriate. In the case where the 3D wearable is a bracelet, forexample, a 6 cubic inch printer may be sufficient. When printing shoeinsoles, however, a larger printer may be required. A majority ofinsoles can be printed with a 1 cubic foot printer, though some largerinsoles require a larger 3-D printer.

Users of the system may take a number of roles. Some users may beadministrators, some may be the intended wearer of an end, 3-D printedproduct, some users may facilitate obtaining input data for the system,and some may be agents working on behalf of any of user type previouslymentioned.

FIG. 2 is a flowchart illustrating a process for generating custom 3Dprinted wearables. In step 202, the mobile device accepts input from auser through the user interface concerning the selection of the type ofwearable the user wants to purchase. In some embodiments, the mobiledevice uses a mobile application, or an application program interface(“API”) that include an appropriate user interface and enablecommunication between the mobile device and external web servers.Wearable examples previously mentioned include shoe insoles, bracelets,rings, bras, and gauge earrings. In addition to these examples, theproduct may be drilled down further into subclasses of wearables. Amongshoe insoles, for example, there can be dress shoe, athletic shoe,walking shoe, and other suitable insoles known in the art. Each subclassof wearable can have construction variations.

In addition to choosing the wearable type and subclass, the user entersin account information such as payment and delivering addressinformation. Some embodiments include social features by which the useris enabled to post the results of a 3D wearable customization process toa social network.

In step 204, the mobile device provides instructions to the user tooperate the camera to capture images of the user (or more precisely, abody part of the user) in a manner which collects the data necessary toprovide a customized wearable for that user. In step 206, the mobiledevice transmits the collected image data to the processing server 24.In step 208, the processing server performs computer vision operationson the image data in order to determine the size, shape, and curvatureof the user (or body part of the user), where applicable to the chosenproduct type.

In step 210, the server application software obtains size and curvaturespecifications from the computer vision process to apply thesespecifications and completes a tessellation model. In some embodiments,the process of applying specifications from computer vision involves theserver application software altering a predetermined set of vertices ona premade tessellation model kit that most closely resembles thewearable the user wants to purchase. In addition, details relating tothe particular chosen wearable type may be applied to the tessellationmodel kit as applicable (e.g., textures or shape modificationspertaining to a subclass of wearable).

In an alternative embodiment, the tessellation file is generated on thefly from the input provided to the system. The tessellation file isinstead generated without premade input through an image processing,computer vision, and machine learning process.

In step 212, the processing server forwards the customized tessellationmodel to the 3D printer. In step 214, the 3D printer prints thecustomized wearable based on the customized tessellation model. In step216, the wearable is delivered to the user. Delivery to the user dependson a number of factors such as the source of the order and thedestination of the order. The system can route the orders to in-house orthird party manufacturers that are located closer to the user(geographic location) in order to facilitate the delivery process.

FIG. 3 is a flowchart illustrating a process for acquiring images of theuser and other data to be used in creating a customized 3D printedwearable. In step 302, upon initiation of the client applicationsoftware in the mobile device, and in a manner similar to step 202 ofFIG. 2, the mobile device prompts for and receives a user selection forwearable type and subclass. In step 304, the client application softwareloads (e.g., from local or remote memory) the necessary instructions toprovide for the wearable type and subclass used to properly direct auser to capture image data of the user's body part or parts asapplicable to the wearable type.

In step 306, the loaded instructions are provided to the user via themobile device's user interface (e.g., via a touchscreen and/or audiospeaker) to facilitate image data capture. The body part imaging mayinvolve acquiring multiple images. The images required may depend on thekind of mobile device used and/or the wearable type selected. Theinstructions provided to the user can include details such as theorientation of the camera and the objects/body parts that should be inframe of the camera's view finder.

In step 308, the mobile device camera captures body part image data. Thebody part image data may come in any of various formats, such as 2Dimages, 3D images, video clips, and body scanner imaging data. In someembodiments, the body part image data may come from a 3rd partyapparatus (such as a body scanner from a doctor's office), a useruploading images via a website.

In step 310, the client application software performs validation on thecaptured image. In some embodiments, the validation is performed by theserver application software. The validation can use object and pixelrecognition software incorporated into either the client or serverapplication software to determine whether or not a given image isacceptable. For example, if the image expected is of the user's foot ina particular orientation, an image of a dog is recognized asunacceptable based on expected variations in pixel color. In the case ofan unacceptable image, in step 312, the current image data is deleted orarchived, and the user is instructed again to capture the correct imagedata. When image data is acceptable, the process moves on to step 314.

In step 314, the application software determines if more images arerequired. If so, the instructions for the next image are presented andthe user is once again expected to orient the mobile device correctly tocapture an acceptable image. Where the application has all the imagedata required, the process continues to step 316.

In step 316, the client application software accepts other usercustomizations that are unconnected to the size, curvature, or shape ofthe wearable. Examples of these other user customizations include color,material, and branding. In step 318, the wearable data collected istransmitted by the mobile device to the processing server.

FIG. 4 is a flowchart illustrating a process by which the mobile deviceinteracts with the user to acquire images of the user. In someembodiments, particular image data is used, and multiple images may berequested. In the shoe insole example, where the user wishes to purchaseinsoles for both feet, five photos of image data for each pair ofinsoles to be fabricated may be requested (e.g., two images of the topof each of the user's feet, two of the inner side of each of the user'sfoot, and a background image of the space behind the side images,without the user's foot). Where the user wishes to obtain only a singleinsole, three images are used. The system does not take the images fromthe foot that will not serve to model a custom insole.

In step 402, the mobile device provides the user with instructions forthe top down views. In some embodiments, where a non-3D camera is used,the instructions include a reference object. In some embodiments, thereference object is a piece of standard-sized paper, such as letter-size(e.g., 8.5×11 inch) or A4 size. Because such paper has well-knowndimensions and is commonly available in almost every home, it can beused as a convenient reference object. Based on length versus widthproportions of the paper, the application software can determineautomatically whether the paper is letter sized, legal size, A4 sized,or other suitable standardized sizes known in the art. Based on thestyle of the paper, the application software has dimensions of knownsize within frame of the image. In other embodiments, the user may beasked to indicate the paper size chosen via the user interface (e.g.,letter-size or A4).

The instructions for the top down image direct the user to find an openfloor space on a hard flat surface (such as wood or tile) in order toavoid warping the paper, thereby casing errors in the predeterminedsizes. The user is instructed to place the paper flush against a wall,stand on the paper and aim the mobile device downward towards the top ofthe user's foot.

In some embodiments, there is additional instruction to put a fold inthe paper such that when placed flush with the wall, the paper does notslide under molding or other wall adornments. Additionally, the mobiledevice user interface includes a level or orientation instruction whichis provided by an accelerometer or gyroscope onboard the mobile device.The level shows the user the acceptable angle at which image data iscaptured.

In some embodiments, no reference object is necessary. Where the mobiledevice includes two cameras, parallax distance measurement between twophotographs may be used to determine a known distance and thereforecalculate sizes of the body part. In some cases it is preferable toperform a number of parallax distance measurements to different pointsbetween the two photographs in order to find comparative distancesbetween those points enabling derivation of additional angular databetween the two photographs. Similarly with the reference object, oncethe image has a first known distance, other sizes within the image (suchas the shape of body parts) may be calculated with mathematicaltechniques known in the art.

Where a single camera is used, additional sensors are utilized toprovide data as necessary. The camera used in conjunction with anaccelerometer, gyroscope, or inertial measurement unit (“IMU”), enablesa similar effect as where there are two cameras. After the first imageis taken, the camera is moved, and the relative movement from the firstposition is tracked by the IMU. The camera then takes a second image.Given information from the IMU the parallax angle between where thefirst image was captured and the second image can be calculated.

The method may be performed with a video clip instead. While the videoclip is captured, the IMU tracks the movement of the mobile devicerelative to a first location. Time stamps between the video clip and theIMU tracking are matched up to identify single frames as static images,the parallax angle between each is solvable and the distance to objectsin the images are identifiable. In some embodiments, the video clip isan uninterrupted clip. The uninterrupted clip may pass around the bodypart capturing image data.

In step 404, the mobile device captures images of the user's foot fromthe top down. Reference is made to the use of a single foot; despitethis, this process is repeated for each foot that the user wishes topurchase an insole for. Later, during image processing, computer vision,and machine learning operations on the processing server, the top downimages are used to determine the length and width of the foot (at morethan one location). Example locations for determining length and widthinclude heel to big toe, heel to little toe, joint of big toehorizontally across, and distance from either side of the first to fifthmetatarsal bones. An additional detail collected from the top down viewis the skin tone of the user's foot.

In step 406, the mobile application or API provides the user withdirections to collect image data for the inner sides of the user's foot.This image is used to later process the curvature of the foot arch. Themobile application or API instructs the user to place the mobile deviceup against a wall and then place a foot into a shaded region of theviewfinder. Based upon predetermined specifications of the model ofmobile device being used, and the orientation of the mobile device(indicated by onboard sensors) the application knows the height of thecamera from the floor. Using a known model of mobile device provides aknown or expected height for the camera lens.

In step 408, the mobile device captures images of the inner side of theuser's foot. Later, during computer vision operations on the processingserver, the inner side images are mapped for the curvature of the user'sfoot arch. Using pixel color differences from the background and thefoot, the computer vision process identifies a number of points (ex:100) from the beginning of the arch to the end. In some embodiments, theserver application software uses the skin tone captured from the topdown images to aid the computer vision process to identify the foot fromthe background.

In some embodiments, additional images of the base of the user's footare also taken. The server application software uses these photos todetermine the depth of the user's foot arch. Without the base imagedata, the depth of the user's foot arch is estimated based on the heightof the foot arch as derived from the inner foot side photos.

In step 410, the mobile device provides instructions to take an imagematching the inner side foot photos, only without a foot. This imageaids the computer vision process in differentiating between thebackground of the image and the foot from prior images. With some apredetermined degree of error tolerance, the difference between theinner side images and the background images should only be the lack of afoot, thus anything in both images would not be the user's foot.

The example in FIG. 4 concerning a user's foot is intended to beillustrative. Many different body parts can be imaged with similar setsof photographs/media, which may vary in angle and/or number based on thebody part.

FIG. 5 is a flowchart illustrating a process for performing computervision on collected user images in order to generate size and curvaturespecifications. FIG. 5 is directed to the example of a foot, thoughother body parts work similarly. The curves of each body part vary, theuse of a foot in this example is represents a complex, curved bodystructure. The steps of FIG. 5 are performed by the server applicationsoftware. In step 502, the processing server receives image data fromthe mobile device. Once received, in step 504 and 506, the processingserver performs computer vision operations on the acquired image data todetermine size and curvature specifications for the user's applicablebody part.

In step 504, the server application software analyzes the image data todetermine distances between known points or objects on the subject'sbody part. Example distances include heel to big toe, heel to littletoe, joint of big toe horizontally across, and distance from either sideof the first to fifth metatarsal bones. This process entails usingpredetermined or calculable distances based on a reference object orcalculated distances with knowledge of camera movement to provide aknown distance (and angle). In some embodiments, the reference objectcan be a piece of standard size paper (such as 8.5″×11″), as mentionedabove. The application software then uses known distances to calculateunknown distances associated with the user's body part based on theimage.

In step 506, the processing server analyzes the image data for body partcurvature. The computer vision process seeks an expected curveassociated with the body part and with the type of wearable selected.Once the curve is found, in step 508, points are plotted along the curvein a coordinate graph (see FIG. 6). Shown in FIG. 6, the coordinategraph 50 includes an X,Y location along the curve in a collection ofpoints 52. Taken together, the collection of points 52 model thecurvature of the body part (here, the arch of a foot).

Returning to FIG. 5, in step 510, the processing server packages thedata collected by the computer vision process into one or more files forinspection. In some embodiments, in step 512, an administrative userconducts an inspection of the output data from the computer visionprocess in relation to the acquired images. If there are obvious errorsin the data (e.g., the curvature graph is of a shape clearlyinconsistent with the curvature of the applicable body part), thegenerated data is deemed to have failed inspection and can be rejected.

In step 514, a user or an administrator may perform a manual edit of theoutput data from computer vision reviewed image data. The systemtransmits a copy of the original images to the user/administrator forediting. The user edits the points and then transmits the edited images.The user only provides a reduced selection of points rather than anentire curvature. If no manual edit occurs, in step 516, the image datais processed through an enhancement process to further improve thedistinction between foot and background. The enhancement process refersto image retouching to improve line clarity through editing sharpness,resolution, or selective color by changing using individual pixel, grouppixel, or vector image edits. The current computer vision reviewedimages are discarded, and the computer vision process is run again. If amanual edit occurs, in step 518, the processing server receives updatedcurvature and/or size specifications.

In step 520, final copies of the size and curvature specifications ofthe user's subject body part are forwarded to a customization engine ofthe server application software on the processing server.

FIG. 7 is a flowchart illustrating a process performed by thecustomization engine for customizing tessellation models. In step 702,the user's selected wearable type and subclass are used to narrow theselection of tessellation model kits from a large number of providedoptions to a select group. For example, if the wearable type is a shoeinsole, all other wearable type tessellation model kits are eliminatedfrom the given printing task process. Then further if the shoe insole isof subclass “running shoe insole,” other shoe insole tessellation modelkits are eliminated. For example, the remaining set of premadetessellation model kits may be those associated with different sizes ofshoe, men's shoes or women's shoes, or of a limited variation of geneticfoot types (ex: normal, narrow, flat, high arch, etc.). The narrowing oftessellation model kits reduces the amount of processing that needs tooccur when customization is applied.

In step 704, the computer vision data, including the size and curvaturespecifications, is imported into the customization engine. In someembodiments, the vision data is roughly categorized, thereby eliminatingirrelevant tessellation model kits. Remaining is a single, determinedmodel kit, which most closely resembles the size and curvaturespecifications. In some embodiments, the tessellation model is builtfrom the ground up on the fly based on the observations in imageprocessing, computer vision and machine learning.

In step 706, the size and curvature specifications are applied to thedetermined model kit. In doing so, predetermined vertices of thedetermined model kit are altered using graph coordinates obtained fromthe computer vision operations.

In step 708, other adjustments can be made to prepare the tessellationmodel for printing. The other adjustments can be either ornamentaland/or functional but are generally unconnected to the measurements ofthe user obtained through computer vision operations. One possibletechnique for adjusting a tessellation model uses so-called “negativenormals.”

FIG. 8 illustrates a technique to create or adjust a complextessellation file that can be used in generating 3D printed wearables asdescribed herein. The technique illustrated in FIG. 8 is referred to asa “negative normal” technique. A 3-D object, model, or lattice is ahollow digital object. The object is constructed from a lattice ofplanar polygons, e.g., triangles or quadrilaterals. Each polygon 802that makes up the lattice of the 3-D object has an outer surface and aninner surface.

The term “normal” in this context refers to a normal vector, i.e., avector that is perpendicular to the surface of a polygon. In a positiveobject, positive normals point outward from the surface polygons andnegative normals point inward from the surface polygons; a negativeobject is the reverse.

Where a 3D object is generated with the inner side and outer surfacereversed such that the inner, negative surface is on the exterior of the3D object, a negative object is created. Where a negative objectintersects with and extends from a positive object, a negative normal iscreated.

A negative normal is treated as an error by conventional 3D printersoftware, which results in the printer not printing the portions of thepositive object that have negative normals. However, negative normalscan be used advantageously to create a 3D printable object with complexgeometry.

FIG. 8 illustrates progressions of a positive object (shoe insole) witha negative object applied as a negative normal. The top-most 3-D object54 is a simple shoe insole positive object, in which the polygons 55that make up the positive shoe insole are visible. The second objectfrom the top 56 is a smoothed version of the positive insole object. Thethird object from the top 58 illustrates the use of a negative normal.

In the third object from the top 58, the positive shoe insole 60intersects with a negative ellipsoid prism 62. The negative ellipsoid 62includes a horizontal upper surface 64 and an angled lower surface 66which contacts the upper surface of the positive shoe insole 60. Thenegative ellipsoid prism 62 further includes a number of verticallyoriented negative cylindrical tubes 68. In the empty space where thetubes 68 are not present, the positive shoe insole is unaffected. Wherethe negative ellipsoid prism intersects the positive shoe insole, thetop surface of the positive shoe insole 60 is reduced in height.

In the fourth object from the top 70, the result of the negative normaltechnique is shown. The positive shoe insole 60 has a number of bumps 72on the topmost surface caused from the removal of the surroundingpolygon lattice. The negative normal technique does not alter thedigital rendering, but only the 3-D printed object. This is because thedigital rendering is a hollow object, whereas the 3-D printed object issolid. Hence, stating that the negative normal “removes” material fromthe positive object is a misnomer, strictly speaking, though it may be aconvenient way of thinking of the technique. More precisely, thematerial is never applied to the 3D printed object in the first place.The bottom-most object 74 in FIG. 8 illustrates the results of thenegative normal technique used a number of times.

If the negative normal technique were not used in the example of FIG. 7,the many bumps 72 of final shoe insole would have to be generatedindividually. The amount of processing resources used to calculate abump which is constructed of many of positive polygonal planes ofvarying size is substantially more than the processing power to create asimple negative shape which has comparatively fewer polygonal planes.

FIG. 9 is a block diagram illustrating a distributed system for thegeneration of customized 3D printed wearables according to the techniqueintroduced here. Embodiments of the system and method herein may bedistributed. The hardware involved may communicate over a networkoperated by different parties, be directly connected operating by thesame party, or any combination thereof.

For example, in some embodiments, the 3D printer 26 is a networklocation and outsourced to a contractor for 3D printing. This contrastswith FIG. 1 in which the 3D printer 26 is directly connected with thebackend processing server 24. Accordingly, instructions are sent to the3D printer 26 over a network. Additionally, in the manual inspectioncomputer 28 may be separate from the backend processing server 24, inboth a physical sense and as an acting entity. For example, the manualinspection computer 28 may be operated by a doctor of a patient who ownsthe mobile device 22. In another configuration, the manual inspectioncomputer 28 may be operated by the same corporate entity that operatesthe processing server 24. In yet another configuration, both the mobiledevice 22 and the manual inspection computer 28 are operated by a doctorof a patient for whom body images/videos are taken.

The above are merely examples —there are multiple combinations ordistributions of actors and hardware. In order to achieve thisdistribution, embodiments of the system introduced here include asoftware interface, such as application program interface (“API”) 54,which is used across the distributed hardware to coordinate inputs andoutputs in order to reach a 3D-printed and delivered wearable. The API54 is instantiated on a number of the hardware objects of the system,and ultimately references databases 56 on the processing server 24.

The database 56 stores body images/videos, associated 3D models of bodyparts, and 3D models of wearable which match the 3D models of the bodyparts. Each of these images/models are indexed by a user or ordernumber. Devices which instantiate the API 54 may call upimages/videos/materials at various points in the wearable generationprocess, provide input, and observe status of said materials. The API 54is able to provide query-able status updates for a given wearable order.In this way, the wearable generation has a great degree of transparencyand modularity.

FIG. 10 is a flowchart illustrating a process for API access at a numberof steps in wearable generation. The process does not necessarily haveto be performed by a single entity. Rather, there are a number ofstarting points and ending points where different entities mayparticipate in the process. The flow chart of FIG. 10 shows inputs onthe left, outputs on the right, and processes down the middle.

Each input (“i” steps) may be performed by one of a number of relevantactors. Alternatively, each process step has a result which is passed tothe next step. Depending on the steps, process steps may includeexternal input, as previous process result as input, or both. Outputs(“o” steps) provide visibility into the process and help provide humanactors information to make decisions in future steps, or to provideentertainment value.

In step 1002, the system obtains image or video data of a part of aliving body. Step 1002 is technically an input as well. For the purposesof FIG. 10, step 1002 refers to obtaining the video or images throughmobile application software 42, making use of a device camera. Themobile application software 42 has an internal (non-API based)connection to the processing server 24.

Alternatively, or in conjunction, the process begins at step 1004 i. Instep 1004 i, the API 54 receives body data, which is subsequentlyprovided to the processing server 24. In this case, the API 54 isinstalled as an extension within other non-operating system software.The API 54 is installed on an imaging device or on a computer whichcontains data collected from an imaging device. This data may vary informat, number of dimensions, and content.

In step 1004, the processing server 24 uses the input (for 1002, 1004 i,or both) to generate a digital body model. The digital body part modelmay come in a number of formats. In some embodiments, the digital bodymodel is a 3D model; in others the digital model is a series of figureswhich model size, shape, and character; in still others the digitalmodel is a number of 2D images including properties and metadata.

The creation of the digital body model occurs first through formatrecognition and then computer vision. The processing server 24 firstidentifies the format of the input received in order to determine howsaid input is processed. Once the format is determined, the method ofcomputer vision is determined. The result of the computer vision processis the digital body model. In some embodiments, such as those alreadyincluding a 3D body model, the computer vision process is skipped.

In step 1004 o, the API 54 exposes the digital body model. Accordingly,users, doctors of users, or other suitable persons relevant to thecreation of an wearable are able to view the data on a number ofhardware platforms. The digital body model is indexed by the user ororder number, and thus searching the database 56 via the API 54 mayreturn the digital body model output.

In step 1006 i, relevant persons are enabled to either provide their ownbody models from 3D imaging devices, and or previously measured figuresthrough the API 54, or cause the API 54 to provide messaging to theprocessing server 24 accepting the body model input of step 1004. Thebody models would come from other existing scanners, or models that weregenerated by the user. The API 54 accepts the pre-generated models asinput and move that input forward in the process as if the usersubmitted body model was the result of step 1004.

In step 1006, the processing server 24 generates a digital wearable 3Dmodel based on the digital body model. The generation of the 3D model ofthe wearable proceeds as methods taught herein have disclosed, or byanother suitable method known in the art. In step 1006 o, the API 54exposes the 3D wearable model. Accordingly, users, doctors of users, orother suitable persons relevant to the creation of a wearable are ableto view the 3D wearable model on a number of hardware platforms. The 3Dwearable model is indexed by the user or order number, and thussearching the database 56 via the API 54 may return the 3D wearablemodel output.

In step 1008 i, relevant persons are enabled through a user interface tocause the API 54 to provide messaging to the processing server 24accepting the 3D wearable model input of step 1006. The output of step1006 o provides the basis for the user's decision. In step 1008, the 3Dwearable model is transmitted to a 3D printer 26 for printing. In someembodiments 3D printing occurs through an additive process, though oneskillful in the art would appreciate 3D printing through reduction.

In step 1010, where there are a number of separately printed componentsof the wearable, these components are assembled. In some embodiments,where the 3D printer 26 is operated by an external printing entity,multiple components are sent to a central location/entity for assembly.In some embodiments the system routes printing and/or assembly to remotelocations based upon delivery address. These steps may be performedclose to the delivery location where assets are available to provide theservice.

At step 1010 o, the processing server 24 uploads an assembly processvideo or time lapse to a host location. Where cameras are available atthe 3D printer 26, or the assembly process, regular image captures orvideos are taken and indexed by user or order number. These imagecaptures or videos may be assembled into a wearable generation video andposted on a web site hosted by the processing server 24, or into anexternal video hosting service (such as YouTube).

In step 1012 i, if an external printer still holds the printed wearable,the wearable is shipped to the central customer management location. Instep 1012, the central customer management ships printed wearable to auser and/or doctor. In some embodiments, elements of the order areprovided to a local asset for delivery to the user.

FIG. 11 is a flowchart illustrating an embodiment for handling a numberof input body data types. While many of these steps refer to theprocessing server 24 as the actor, multiple sources may actually performthese steps as illustrated by FIG. 10. In step 1102, body data isobtained. As described above, the character of the body data may vary.Examples include: 2D/3D images/video with various levels of penetrationand light exposure/usage (e.g. conventional photography/video; infraredimaging/video data; confocal microscopy data; lightfield imaging data;or ultrasound imaging data). This data may come from a number ofsources. In step 1104 the body data is transmitted to the processingserver 24.

Once transmitted, the format of the data is determined. In step 1106,the processing server 24 categorizes the data as video or still. Wherethe data is a still, in step 1108 the data is categorized asphotograph(s) or scanner models. A scanner model refers to a 3D modelgenerated by a body scanner.

In step 1110, where the body data is a video, the processing server 24parses frames (stills) of the video and matches to model still images.The matching is performed with pixel and object comparisons. Objects areidentified in the frames based on pixels. Pixels or objects are matchedto pixels or objects in the model images. In step 1112, the processingserver 24 extracts the frames that best match the model images from thevideo as stills. In step 1114 the processing server 24 converts theframes into image files.

In step 1116, where a still photograph is detected, the photograph isconverted into an image format that computer vision may be performed on.This is relevant in the case of, for example, images that are not takenin the visible spectrum (such as infrared or ultrasound images). In thisstep objects are identified from the pixels in the image file. In step1118, images are matched to model images similarly to in step 1110.However, depending on where the body data photographs originated from,the photographs may already match model images (e.g., where photocapture instructions are followed prior to transmitting the body data).

In step 1120, regardless of the original body data input, the currentproduct of the process with be in the same image data format after theconversions in steps 1114 and 1118. Accordingly, the processing server24 runs the computer vision process on the images. The computer visionprocess identifies characteristics of the photographed/recorded bodypart. In step 1122, the processing server 24 generates a digital bodymodel. A digital model may take a number of forms, one of whichoriginates from a body scanner in step 1108 without additionalmodification. Other digital body models are descriptive numbers inparticular parameter fields. Regardless of the type of digital bodymodel, in step 1124, the processing server 24 generates a wearabletessellation file that corresponds to the digital body model. In step1126, a 3D printer prints that tessellation file.

FIG. 12 is a flowchart illustrating wearable generation includingconcurrent computer vision and machine learning processes. FIG. 12 is adetailed look at step 1004 of FIG. 10 and some surrounding steps. Thesteps of FIG. 12 are generally performed by the processing poweravailable within the entire system. However, the processing power may bedistributed across a number of devices and servers. For example, somesteps may be performed by a mobile device such as a smart phone whileothers are performed by a cloud server.

In step 1200, input body part image data is provided to the system. Theinput data may be provided in any of various ways (e.g. through directupload from smartphone applications, web uploads, API uploads, partnerapplication uploads, etc.). In step 1202, the uploaded body part imagedata is processed to obtain a uniform image format for furtherprocessing. FIG. 11 details portions of the image pre-processing.

Once the body image input data is in a uniform data format, the methodproceeds to steps 1204 and 1206. In steps 1204 and 1206, the systemattempts to detect a body part in the subject images. This is performedboth through computer vision and machine learning. Prior observationsand models (e.g. hidden Markov model) influence the machine learningoperation. The detection of a particular body part enables the system todetermine the type of product that is most relevant. In someembodiments, the system performs the body part identification initiallyto enable the user to select a product type choice (e.g. footwearinsole, bra, earphones, gloves, etc.).

In step 1208, the system checks whether a body part was, in fact,detected. Where a body part was detected, the method proceeds, whereaswhere a body part was not detected, the method skips to observationalsteps to update the machine learning models. The user interface willadditionally signal the user and the user may initiate the method againform the beginning.

In some embodiments, steps 1204-1208 are processed by local mobiledevices owned by a user. Because these steps are often performed beforethe user selects a product type (steps identify body part and productchosen after body part identification), some efficiency may be gained byusing local processing power as opposed to transmitting the data priorto making the body part identification.

Further, the product type selection user experience enables the systemto “hide” the data transmission from the mobile device to the cloudserver. Once a body part is identified, the user begins selecting aproduct style, types and sub-types. While the user is engaged in theseoperations, the body part image data is uploaded from the mobile deviceto the cloud server, and the user experience is not stalled.

In steps 1210 and 1212, the system performs image segmentation usingcomputer vision and machine learning. Prior observations and models(e.g. hidden Markov model) influence the machine learning operation.Image segmentation is used to identify particular elements in an imagesuch as differentiating the body part from the background, as well asdifferentiating different curves and surfaces of the body part.

In step 1214, the system identifies regions of the body part which arerelevant for curvature of 3-D models (such as tessellation files). Insteps 1216 and 1218, the system performs point extraction using computervision and machine learning. Prior observations and models (e.g. hiddenMarkov model) influence the machine learning operation. The pointsextraction process generates a number of points along curves identifiedthrough segmentation. Each of these points is assigned a point in either2-D or 3-D space. Measurements for point coordinates are provided basedon solved distances using reference objects, or solving through imageanalysis.

In step 1220, the extracted data points are assembled into usable datafor 3-D model generation or tessellation file generation. In step 1222,the data points and body part images undergo post-processing. In step1224, The system adds the data points and body part image data to thetotal observations. In step 1226, the system enables users, oradministrators to do an audit review. This step is detailed in aboveparagraphs. At this point the data points are delivered to modelgeneration and the rest of the 3-D printing process continuesseparately.

After steps 1220-1226, the data is added to a database. The process ofFIG. 12 continues with an assessment and learning phase. In step 1228,the system reviews and performs a performance assessment of the process.In step 1230, the machine learning engine of the system updates theobservations from the database and the performance assessment. If theprocess continues, in step 1234, the machine leaning models are updated.The updated machine learning models are recycled into use into steps1204, 1210, and 1216 for subsequent users.

FIG. 13 is a flowchart illustrating distance measurement in images takenfrom single lens 2-D cameras. Where the mobile device includes a singlecamera and an inertial measurement unit (IMU) parallax distancemeasurement between two photographs may be used to determine a knowndistance and therefore calculate sizes of the body part. Similarly withthe reference object, once the image has a first known distance, othersizes within the image (such as the shape of body parts) may becalculated with mathematical techniques known in the art.

In step 1302, a first image is taken in a first position. In step 1304the camera is moved, and the IMU tracks the relative movement from thefirst position to a second position. In step 1306, The camera then takesa second image.

The method may be performed with a video clip as well. While the videoclip is captured, the IMU tracks the movement of the mobile devicerelative to a first location. Time stamps between the video clip and theIMU tracking are matched up to identify single frames as static images.In step 1308, the system identifies regions of interest in the images.This is explained in FIG. 12 at step 1214.

In step 1310, given information from the IMU the system calculates theparallax angle between where the first image was captured and the secondimage. In step 1312, the system calculates distance to the region orpoint of interest based on the parallax angles and distances between thefirst and second position. In step 1514, the system is able to usegeometric math to solve for the distance between a number of distanceswithin each image. These distances are used to provide coordinates to anumber of points in images, and then later used to develop 3-D models ofobjects within the images.

FIG. 14 is a graphic illustration of a customized computer mousegenerated through body scanning. In addition to wearables, the processestaught in this disclosure may be used to generate items interfacing withthe contours of a user's body as well. One example of such an item is acomputer mouse 76. Mouse peripherals are designed to have a significantinterface with the human body, specifically, a human hand andfingertips. Where body image data is collected on a user's fingers, thesystem may generate a computer mouse 76 with matching fingerindentations 78 custom printed for a particular user.

FIG. 15 is a graphic illustration of a customized ear headphonegenerated through body scanning. Another example of a wearable that canbe custom designed in the disclosed system is a headphone 80, theheadphone has a speaker enclosure 82 which is custom formed to an earcavity. Using body image data of a person's ear enables customgeneration of fitted ear cavity speakers 82.

FIG. 16 is a graphic illustration of an assortment of wearables that aregenerated via body imaging followed by 3-D printing. The illustrationsin FIG. 16 are intended to be illustrative of 3D-printable wearablesthat conform to a body part of the wearer. Examples include a bra 84, ahelmet 86, a brace 88, or goggles 90. There are many possible wearablessuited for many purposes.

FIG. 17 is a graphic illustration of customized eye glasses generatedthrough body scanning. Another example of a wearable that can be customdesigned in the disclosed system are eye glasses. A number of segmentsmay be customized to a user's body.

FIG. 18 is a graphic illustration of a customized brace generatedthrough body scanning. Another example of a wearable that can be customdesigned in the disclosed system is a brace type orthotic. A number ofsegments may be customized to a user's body.

Although the invention is described herein with reference to thepreferred embodiment, one skilled in the art will readily appreciatethat other applications may be substituted for those set forth hereinwithout departing from the spirit and scope of the present invention.Accordingly, the invention should only be limited by the claims includedbelow.

1. A method comprising: receiving, through an application programinterface (API), body part image data of a part of a living body;determining a set of physical dimensions of the body part by using aplurality of images of the body part image data as parallax viewpoints,wherein a position of each parallax viewpoint and a set of points on thepart of the living body identified via pixel recognition are used toidentify a depth, and wherein the depth is used to derive the set ofphysical dimensions; generating a digital model of the body part basedon the set of physical dimensions; generating a 3D model of a wearablecorresponding to the body part based on the digital model; and sendingprint instructions to a 3D printer including information associated withprinting the 3D model of the wearable to a physical wearable.
 2. Themethod of claim 1, further comprising: receiving a message, by the API,from a user, the message accepting the 3D model of the wearable; andprinting, by a 3D printer, a wearable corresponding to the 3D model ofthe wearable.
 3. The method of claim 1, further comprising: receiving,by a mobile device application, part images of the living body from amobile device camera; filtering, by the mobile device application, thebody part images into accepted images and rejected images based upon acomputer vision comparison to a set of expected images; andcommunicating, by the mobile device application, accepted images to theAPI as the body part image data.
 4. The method of claim 1, furthercomprising: receiving, by a mobile device application, images of theliving body from a mobile device camera, the images including apredetermined reference object having known dimensions; and whereindetermining the set of physical dimensions further includes a comparisonof the body part to the reference object.
 5. The method of claim 1,wherein the part of the living body is a foot, and the 3D model of awearable is a footwear insert.
 6. The method of claim 5, furthercomprising: receiving, by a mobile device application, foot images ofthe living body from a mobile device camera, the foot images including apredetermined reference object having known dimensions; and whereindetermining the set of physical dimensions further includes a comparisonof the body part to the reference object.
 7. The method of claim 6,wherein the reference object is a sheet of standardized sized paper. 8.The method of claim 3, further comprising: issuing instructions, by themobile device application, to a user of an associated mobile devicedepicting expected positioning of the body part.
 9. The method of claim1, wherein the body part image data comprises at least one of: infraredimaging data; confocal microscopy data; lightfield imaging data; orultrasound imaging data.
 10. The method of claim 1, further comprising:collecting, by a mobile application of a mobile device, video data of apart of a living body, the video data including a video clip wherein afilm perspective rotates about the part of the living body; andextracting image frames from the video data, the image frames includingthe part of the living body captured at a number of perspectives,wherein the image frames are determined by comparing and matching eachframe of the video data to reference frames, wherein the referenceframes preexist the video data.
 11. The method of claim 1, furthercomprising: printing, by a 3D printer, a wearable corresponding to the3D model of the wearable; generating a time-lapse video depicting saidprinting step; and transmitting the time-lapse video to a web host forhosting.
 12. The method of claim 1, wherein the received body image datais of a first format, the first format is computer readable and one of anumber of predetermined formats, and wherein the generating a digitalmodel of the body part step further comprises: identifying the firstformat from the number of predetermined formats; determining a subset ofdata to extract from the body image data of the first format to use inconverting at least a portion of the body image data to a result format;extracting the subset of the body image data from the body image data;and converting the subset of the body image data to the result format.13. A system comprising: a network-connected server configured toprocess body part image data of a part of a living body, and to generatea 3D model of a wearable corresponding to the body part based on thebody part image data, wherein a set of physical dimensions of the bodypart is calculated using a plurality of images of the body part imagedata as parallax viewpoints, wherein a position of each parallaxviewpoint and a set of points on the part of the living body identifiedvia pixel recognition are used to identify a depth, and wherein thedepth is used to derive the set of physical dimensions; and an APIinstantiated on a number of devices external to the server andconfigured to communicate with the server, the API configured to receivethe body part image data on a first device of the external devices andto expose the corresponding 3D model of the wearable to at least asubset of the number of external devices.
 14. The system of claim 13,further comprising: a communication interface configured to output the3D model of the wearable to a 3D printer to print the wearable from the3D model of the wearable.
 15. The system of claim 13, furthercomprising: a mobile device application instanced on a mobile device,the mobile device application in communication with the web server via amobile device network communicator and is integrated with the API, themobile device application configured to interface with a mobile devicecamera and capture body part image data.
 16. The system of claim 14,further comprising: a video server that hosts time-lapse videos that areaccessible from the Internet.
 17. A method comprising: receiving,through an application program interface (API), body part image data ofa part of a living body; determining a set of physical dimensions of thebody part without use of a reference object in the body part image data,wherein the body part image data is derived from a single-lens visiblelight photodetector; generating a digital model of the body part basedon the set of physical dimensions; generating a 3D model of a wearablecorresponding to the body part based on the digital model; and sendingprint instructions to a 3D printer including information associated withprinting the 3D model of the wearable to a physical wearable.
 18. Themethod of claim 17, further comprising: receiving, by a mobile deviceapplication, part images of the living body from a mobile device camera;filtering, by the mobile device application, the body part images intoaccepted images and rejected images based upon a computer visioncomparison to a set of expected images; and communicating, by the mobiledevice application, accepted images to the API as the body part imagedata.
 19. The method of claim 18, further comprising: issuinginstructions, by the mobile device application, to a user of anassociated mobile device depicting expected positioning of the bodypart.
 20. The method of claim 17, further comprising: collecting, by amobile application of a mobile device, video data of a part of a livingbody, the video data including a video clip wherein a film perspectiverotates about the part of the living body; and extracting image framesfrom the video data, the image frames including the part of the livingbody captured at a number of perspectives, wherein the image frames aredetermined by comparing and matching each frame of the video data toreference frames, wherein the reference frames preexist the video data.